In response to unprecedented growth in the amount of data generated in modern computing environments, data storage systems are leveraging object storage paradigms. Object storage paradigms store data in chunks and deduplicate the chunks to minimize or eliminate redundant storage of data.
Using object storage paradigms entails a certain amount of overhead. For example, when storing a file, the file must be broken into the chunks and deduplicated such that the deduplicated chunks can be located and reassembled for subsequent retrieval. Managing object storage requires data structures and processing logic, both of which result in overhead when storing and retrieving files.
When used to store large files, the amount of overhead to implement an object storage paradigm is generally small relative to the large amount of redundant data that can be eliminated. When used to store small files, however, the amount of overhead can be even larger than the data being stored. For this reason, data storage systems that use object storage paradigms to store small files often cost more in storage and processing than is saved.